Skip to content

Rabbitmq topology resources#16705

Open
tjwald wants to merge 17 commits intomicrosoft:mainfrom
tjwald:rabbitmq-topology-resources
Open

Rabbitmq topology resources#16705
tjwald wants to merge 17 commits intomicrosoft:mainfrom
tjwald:rabbitmq-topology-resources

Conversation

@tjwald
Copy link
Copy Markdown

@tjwald tjwald commented May 3, 2026

Description

Adds first-class child-resource support to the RabbitMQ hosting integration: virtual hosts, queues, exchanges, bindings, shovels, and policies. App authors can declare their full RabbitMQ topology in the AppHost, and Aspire will provision it against the running broker and surface per-resource health on the dashboard.

var rabbit = builder.AddRabbitMQ("rabbit").WithManagementPlugin();

var vhost = rabbit.AddVirtualHost("events");
var queue = vhost.AddQueue("orders");
var exchange = vhost.AddExchange("orders-ex", RabbitMQExchangeType.Topic)
                    .WithBinding(queue, routingKey: "order.*");

vhost.AddShovel("archive", queue, vhost.AddQueue("orders-archive"));
vhost.AddPolicy("ttl", pattern: "^orders.*", applyTo: RabbitMQPolicyApplyTo.Queues)
     .WithProperties(p => p.MessageTtl = TimeSpan.FromMinutes(10));

How it works

  • Provisioning — runs at ResourceReadyEvent in three phases per virtual host: vhost create → declare queues/exchanges/shovels/policies in parallel → apply exchange bindings. Failures are isolated per resource (one bad queue doesn't take down its siblings); a vhost-level failure cascade-faults its children. Communicates with the broker over AMQP for declares, and the management API for vhosts, shovels, and policies.
  • Health checks — every child resource gets its own health check that reports Degraded while provisioning is in flight, Unhealthy on a faulted apply, and otherwise runs a live probe (passive declare for queues/exchanges, status poll for shovels, GET for policies).
  • PluginsWithPlugin(RabbitMQPlugin) enables a known plugin; WithManagementPlugin() and AddShovel auto-enable the plugins they need.

Fixes #16579

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes (135 tests across builder shapes, provisioner phases, health checks, policies, plugins, and public-API validation)
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
    • No
  • Does the change require an update in our Aspire docs?
    • Yes
      • Link to aspire.dev issue: will file once the API surface is signed off.
    • No

Open follow-ups (not in this PR)

  • TaskCompletionSource exposed on the provisioning interface — internal IRabbitMQProvisionable currently exposes the raw TCS, which means anything holding the interface can mutate provisioning state. Open API design question for maintainers.
  • Cross-broker shovelsAddShovel only accepts a source and destination on the same RabbitMQ server. Multiple local servers, or a local-to-remote shovel, are not yet supported. Can be added later by accepting a connection-string parameter for the remote endpoint.
  • Publish & deployment support — child resources are run-mode only. Manifest emission and deployer integration (so the topology survives aspire publish/deploy) is intentionally deferred.
  • Playgroundplayground/rabbitmq/ AppHost exercising the full new surface end-to-end is designed but not in this PR.
  • DAG Evaluation of provisioning - I could have implemented a full DAG flow for provisioning but that seemed exccesive and premature optimization. We can optimize this later. For example, if an exchange and a queue are provisioned we can provision the binding before we wait for all other queues to provision, which is the current behavior.

tjwald added 13 commits May 3, 2026 11:16
…erver VirtualHosts list

- Add RabbitMQQueueType, RabbitMQExchangeType, RabbitMQDestinationKind,
  RabbitMQShovelAckMode, RabbitMQPlugin enums
- Add IRabbitMQDestination marker interface (EntityName, VirtualHost, Kind)
- Add internal RabbitMQPluginAnnotation : IResourceAnnotation
- Extend RabbitMQServerResource with internal VirtualHosts list and
  HasPluginFileCallback flag
…d provisioning client

Resource model:
- RabbitMQVirtualHostResource: IResourceWithParent<RabbitMQServerResource>,
  IResourceWithConnectionString, vhost-aware AMQP URI, internal TopologyReady TCS,
  internal ApplyAsync(IRabbitMQProvisioningClient)
- RabbitMQQueueResource: IResourceWithParent<RabbitMQVirtualHostResource>,
  IRabbitMQDestination, Durable/Exclusive/AutoDelete/QueueType/Arguments
- RabbitMQExchangeResource: IRabbitMQDestination, Type/Durable/AutoDelete/Arguments,
  internal List<RabbitMQBinding>, ApplyAsync + ApplyBindingsAsync
- RabbitMQBinding: [AspireDto] POCO (Destination, RoutingKey, Arguments)
- RabbitMQShovelResource: IResourceWithParent<RabbitMQVirtualHostResource>,
  Source/Destination endpoints, AckMode, ReconnectDelay, SrcDeleteAfter
- RabbitMQShovelEndpoint: [AspireDto] wrapping IRabbitMQDestination

Provisioning client:
- IRabbitMQProvisioningClient: 9-method internal interface (AMQP + HTTP)
- RabbitMQProvisioningClient: sync ctor, lazy GetOrCreateChannelAsync /
  GetOrCreateHttpClientAsync, per-vhost IChannel cache, normalises errors
- RabbitMQShovelDefinition: internal JSON record for PUT /api/parameters/shovel
- RabbitMQTopologyProvisioner: pure orchestration — vhost → exchanges → queues
  → bindings → shovels → TopologyReady.TrySetResult()
…ge, Shovel, Plugin

New builder methods (all decorated with [AspireExport] or [AspireExportIgnore]):
- AddVirtualHost: auto-enables management plugin for non-'/' vhosts
- GetOrAddDefaultVirtualHost: lazy idempotent '/' vhost creation
- AddQueue (vhost + server overloads): creates RabbitMQQueueResource
- AddExchange (vhost + server overloads): creates RabbitMQExchangeResource
- WithProperties (queue/exchange/shovel): ATS-exported configure callbacks
- WithBinding<TDest>: same-vhost validation + WithRelationship
- AddShovel<TSrc,TDest> (vhost + server overloads): auto-enables shovel plugins
- WithPlugin(RabbitMQPlugin) + WithPlugin(string): lazy enabled_plugins file
  generation via WithContainerFiles; management defaults always included

DI wiring:
- Register IRabbitMQProvisioningClient as keyed singleton per server
- Subscribe ResourceReadyEvent → RabbitMQTopologyProvisioner (no-op when empty)
- Replace ad-hoc IConnection? closure with client reuse for server health check

Per-child health checks:
- Vhost: AMQP connect on vhost after TopologyReady
- Queue: QueueExistsAsync after TopologyReady
- Exchange: ExchangeExistsAsync after TopologyReady
- Shovel: GetShovelStateAsync == 'running' after TopologyReady
…ins, and public API

New test files:
- TestServices/FakeRabbitMQProvisioningClient: records all calls; shared by
  unit tests and provisioner ordering tests
- AddRabbitMQChildResourcesTests: parent/child wiring, idempotent default vhost,
  cross-vhost validation, connection-string expressions
- RabbitMQTopologyProvisionerTests: call ordering (vhost→exchanges→queues→
  bindings→shovels), TopologyReady completion, exception propagation
- RabbitMQPluginTests: enabled_plugins generation, dedup, management defaults
  always included, enum→string mapping, auto-enable matrix

Extended test files:
- ConnectionPropertiesTests: snapshot test extended with vhost/queue/exchange
  manifests (snapshot generated on first run)
- RabbitMQPublicApiTests: null/empty argument guards for all new builder methods
  (AddVirtualHost, AddQueue, AddExchange, WithProperties, WithBinding, WithPlugin)
…s; add design plan

README additions:
- Virtual host, queue, exchange, binding, shovel usage examples
- Server-level convenience overloads section
- WithProperties section
- Plugin customization section with auto-enable matrix table
- Connection properties table for VirtualHost, Queue, Exchange resources

plans/rabbitmq-child-resources.md: full approved design document covering
resource model, builder API, provisioning architecture, WaitFor semantics,
plugin file generation, validation rules, and implementation backlog
Replace the single vhost-level TopologyReady TCS with per-resource
ProvisioningComplete TCSs so that partial provisioning failures are
attributed only to the affected resource, not to all siblings.

Changes:
- IRabbitMQProvisionable: add Name, ProvisioningComplete, HealthDependencies
  (default []), and ProbeAsync (default Healthy) so each resource owns its
  own health semantics without pulling HealthChecks into the model layer.
- RabbitMQProbeResult: new internal record struct keeping resource classes
  free of Microsoft.Extensions.Diagnostics.HealthChecks.
- IRabbitMQProvisioningClient / RabbitMQProvisioningClient: add CanConnectAsync
  used by the vhost probe, eliminating the concrete-type cast that existed
  in the old vhost health-check registration.
- RabbitMQVirtualHostResource: remove TopologyReady; add ProvisioningComplete;
  ProbeAsync -> CanConnectAsync.
- RabbitMQQueueResource: add ProvisioningComplete; ProbeAsync -> QueueExistsAsync;
  HealthDependencies stub (AppliedPolicies, populated by future policy feature).
- RabbitMQExchangeResource: same pattern; ProbeAsync -> ExchangeExistsAsync.
- RabbitMQShovelResource: add ProvisioningComplete; ProbeAsync -> GetShovelStateAsync.
- RabbitMQProvisionableHealthCheck: single IHealthCheck implementation shared
  by all child resources: await self TCS -> await each dep TCS -> ProbeAsync.
- RabbitMQTopologyProvisioner: rewritten with ApplyAndSignalAsync helper.
  Phase-1 vhost failure cascades to all children (only legitimate cascade).
  Phase-2/3 failures are captured per-entity without short-circuiting siblings.
  Exchange ProvisioningComplete fires after both declare AND bindings.
- RabbitMQBuilderExtensions: single private WithProvisionableHealthCheck<T>
  helper replaces four inline HealthCheckRegistration blocks; all Add* call
  sites become identical one-liners. File-scoped RabbitMQEntityHealthCheck
  deleted.
- Tests: FakeRabbitMQProvisioningClient gains CanConnectAsync and per-entity
  failure injection (FailQueueNames, FailExchangeNames, FailBindingSourceExchangeNames).
  FailingFakeRabbitMQProvisioningClient gains CanConnectAsync. Provisioner
  tests updated to use ProvisioningComplete and extended with isolation cases
  (queue-B fails, binding fails, vhost fails). New RabbitMQProvisionableHealthCheckTests
  covers all health-check stages and per-resource ProbeAsync.
docs/specs/rabbitmq-health-checks.md: contributor-facing design spec
covering the guiding principle, key design decisions (per-resource TCS,
two-stage check, resource-owned semantics, probe result type, binding/shovel/
policy attribution rules), and extension guidance for adding new resource types.

src/Aspire.Hosting.RabbitMQ/README.md: add 'Health checks' section with the
user-facing contract — failure isolation, the vhost cascade, and binding
attribution — without volatile implementation detail.
- RabbitMQPolicyApplyTo enum (Queues, Exchanges, All)
- RabbitMQPolicyResource: IResourceWithParent<RabbitMQVirtualHostResource>,
  IRabbitMQProvisionable; props PolicyName/Pattern/ApplyTo/Definition/Priority;
  internal AppliesTo(entityName, kind) with cached compiled Regex;
  default HealthDependencies=[]; default ProbeAsync=Healthy
- RabbitMQPolicyDefinition: internal JSON record for PUT /api/policies/{vhost}/{name}
- RabbitMQVirtualHostResource: add internal Policies list; cascade policy TCS
  faults in vhost-failure path
- RabbitMQQueueResource + RabbitMQExchangeResource: add AppliedPolicies list;
  explicit-impl HealthDependencies => AppliedPolicies
- IRabbitMQProvisioningClient + RabbitMQProvisioningClient: add PutPolicyAsync
- RabbitMQTopologyProvisioner: Phase 1.5 applies policies after vhost creation
  and before entity declaration; cascade policy TCS faults on vhost failure
- RabbitMQBuilderExtensions: AddPolicy (vhost + server overloads), WithProperties
  for policy; BeforeStartEvent handler resolves pattern matches lazily;
  internal ResolveAndApplyPolicyMatches helper for testability
- FakeRabbitMQProvisioningClient: add PutPolicyAsync stub + FailPolicyNames
- FailingFakeRabbitMQProvisioningClient: add PutPolicyAsync stub
- RabbitMQProvisionableHealthCheckTests: add PutPolicyAsync to FixedShovelStateClient
- RabbitMQPolicyTests: 20 tests covering wiring, lazy matching, provisioner
  ordering, health dependency isolation
- README: add Policies section
…esourceLoggerService logging, ProvisionedName rename, policy health check + probe, vhost existence check, cross-vhost shovels, enabled_plugins cleanup, Queue.Arguments docs
Copilot AI review requested due to automatic review settings May 3, 2026 08:25
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 3, 2026

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 16705

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 16705"

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR extends the RabbitMQ Aspire integration to model and provision broker topology (vhosts, queues, exchanges, bindings, shovels, and policies) as first-class resources, with per-resource health checks and improved plugin handling.

Changes:

  • Added new RabbitMQ topology resource types and a topology provisioner that applies them in ordered phases and isolates failures per resource.
  • Introduced a shared provisionable health-check implementation that waits for provisioning signals, dependency signals (e.g., policies), and performs a live broker probe.
  • Added plugin customization support that generates an enabled_plugins file, plus extensive new unit/functional tests and updated docs/snapshots.

Reviewed changes

Copilot reviewed 41 out of 41 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
src/Aspire.Hosting.RabbitMQ/RabbitMQBuilderExtensions.cs Adds new topology APIs (vhost/queue/exchange/shovel/policy), provisioning wiring, plugin handling, and health-check registration.
src/Aspire.Hosting.RabbitMQ/Provisioning/* Implements provisioning client + topology provisioner + shared provisionable health check and DTOs.
src/Aspire.Hosting.RabbitMQ/*Resource.cs + enums Adds new resource model types and supporting enums/interfaces for topology + probing/dependencies.
tests/Aspire.Hosting.RabbitMQ.Tests/* Adds test fakes and broad coverage for provisioning order, failure isolation, health checks, plugins, and connection properties.
src/Aspire.Hosting.RabbitMQ/README.md + docs/specs/rabbitmq-health-checks.md Documents new topology and health-check behavior and the intended design.
tests/...Snapshots/*.verified.json Updates snapshot to include new connection properties for vhost/queue/exchange references.

Comment thread src/Aspire.Hosting.RabbitMQ/IRabbitMQDestination.cs Outdated
Comment thread src/Aspire.Hosting.RabbitMQ/RabbitMQBuilderExtensions.cs
Comment thread src/Aspire.Hosting.RabbitMQ/RabbitMQBuilderExtensions.cs
Comment thread src/Aspire.Hosting.RabbitMQ/README.md Outdated
Comment thread docs/specs/rabbitmq-health-checks.md Outdated
- Split IRabbitMQDestination into public properties-only interface + internal IRabbitMQBindableDestination with BindAsync
- Validate non-null wire-name overrides (virtualHostName, queueName, exchangeName, shovelName, policyName) with ThrowIfNullOrWhiteSpace
- Sort enabled_plugins file content deterministically with OrderBy(Ordinal) to avoid nondeterministic diffs
- Dispose stale connection/channel before replacing in GetOrCreateChannelAsync to fix connection leak
- Remove misleading 'add policy after queues' ordering constraint from README (matching is resolved at model-freeze)
- Remove '(planned)' label from policy-cascade section in health-checks spec (feature is implemented)
@tjwald
Copy link
Copy Markdown
Author

tjwald commented May 3, 2026

Addressed Copilot review (commit b09f7f5 on rabbitmq-topology-resources). 135/135 unit tests still passing.

Fixed (5 of 6 items):

  • r3177844025IRabbitMQDestination internal member trap: Split into a public properties-only IRabbitMQDestination + a new internal interface IRabbitMQBindableDestination : IRabbitMQDestination that carries BindAsync. Both concrete types (RabbitMQQueueResource, RabbitMQExchangeResource) implement the internal interface; the call site in ApplyBindingsAsync casts to IRabbitMQBindableDestination. This is option (c) from Copilot's suggestion.

  • r3177844033 — Wire-name override validation: Added ArgumentException.ThrowIfNullOrWhiteSpace guards for all optional wire-name parameters (virtualHostName, queueName, exchangeName, shovelName, policyName) when the caller provides a non-null value.

  • r3177844035enabled_plugins ordering (partial fix): Changed from ToHashSet (nondeterministic) to Distinct + OrderBy(Ordinal) so the file content is stable across runs. Pushing back on the quoting suggestion: all RabbitMQ plugin names in the official distribution use only lowercase letters, digits, and underscores — they are valid Erlang atoms without quoting. The canonical form in the official RabbitMQ Docker image is unquoted (e.g. [rabbitmq_management,rabbitmq_shovel].). Adding single-quotes would diverge from that convention without a real safety benefit for the plugin names we generate.

  • r3177844041 — Connection/channel leak on reconnect: Added TryRemove + try/catch dispose of the stale (IConnection, IChannel) pair before creating a new one in GetOrCreateChannelAsync.

  • r3177844043 — README misleading policy ordering: Removed the "Call AddPolicy after all queues and exchanges" instruction. The README now correctly states that order does not matter — matching is resolved at model-freeze time via the BeforeStartEvent handler.

  • r3177844055 — Spec "(planned)" label: Removed "(planned)" from the "Policy failures cascade to matching queues/exchanges" section heading and updated the body to reflect that the feature is implemented and tested.

tjwald added 2 commits May 8, 2026 11:06
…d public API by consolidating similar concepts.
…tate

Replace the externally-signaled TaskCompletionSource on IRabbitMQProvisionable
with a read-only Task ProvisionedTask. Each resource now owns its TCS privately
and signals it from within ApplyAsync (and ApplyBindingsAsync for exchanges).

Resources also publish their own Aspire lifecycle state (Starting/Running/
FailedToStart) via ResourceNotificationService passed into ApplyAsync, so
child resources appear in the dashboard resource list with correct state.

Key changes:
- IRabbitMQProvisionable: TaskCompletionSource ProvisioningComplete ->
  Task ProvisionedTask; ApplyAsync gains ResourceNotificationService and
  ResourceLoggerService parameters; implementations must not throw
- All five resource classes: TCS is private readonly; ApplyAsync publishes
  Starting -> Running/FailedToStart and signals TCS internally
- RabbitMQExchangeResource: ApplyAsync publishes Running after declaration
  (exchange is live); ApplyBindingsAsync signals TCS after bindings
- RabbitMQTopologyProvisioner: no TCS manipulation anywhere; cascade-fault
  block deleted (children stay Degraded/Starting when vhost fails);
  provisioner collapses to a single ProvisionVirtualHostAsync method
- RabbitMQProvisionableHealthCheck: ProvisioningComplete.Task ->
  ProvisionedTask (two property references)
- docs/specs/rabbitmq-health-checks.md: updated with two-signal model,
  lifecycle/health state matrix, and updated extension guidance
Copy link
Copy Markdown
Author

@tjwald tjwald left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self review for agent:

  • docs are verbose
  • provisioning interface is not ideal

Comment thread src/Aspire.Hosting.RabbitMQ/Provisioning/RabbitMQProvisionableHealthCheck.cs Outdated

namespace Aspire.Hosting.RabbitMQ.Provisioning;

internal sealed class RabbitMQProvisioningClient : IRabbitMQProvisioningClient
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extract a file scoped helper class for connection management from the RabbitMQProvisioningClient that implements IAsyncDisposable and migrate related fields, methods and behavior there.

Comment thread src/Aspire.Hosting.RabbitMQ/Provisioning/RabbitMQTopologyProvisioner.cs Outdated
Comment thread src/Aspire.Hosting.RabbitMQ/RabbitMQExchangeArguments.cs Outdated
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs in this file add nothing

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation in this file won't be needed if we add in the same file a static helper class with the switch case for the names.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is an annotation really the correct approach here? Why not have a HashSet on the only type that needs this instead?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs on the enum values are redundant.

…prove XML docs

- Add RabbitMQPlugin.WebDispatch enum value and move plugin name mapping
  into a new internal RabbitMQPluginNames.ToPluginName() extension method,
  keeping domain knowledge close to the enum
- Use RabbitMQPlugin.WebDispatch instead of string literal in WithManagementPlugin
- Remove HasAnyValue() from RabbitMQQueueArguments and RabbitMQExchangeArguments;
  inline the checks in RabbitMQPolicyResource.ApplyAsync
- Improve XML documentation across all source files: tighten summaries,
  move inline remarks into <remarks> tags, remove docs that explain RabbitMQ
  internals rather than our API, and improve enum member descriptions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RabbitMQ Hosting Integration - Add more functionality

2 participants